Currently deployed network intrusion prevention systems, such as, for example, Snort, detect intrusions and malicious activity by matching real-time network packets against a library of intrusion signatures or patterns based on known attack vectors. These patterns can be defined only after an intrusion occurs and has been identified. Writing and applying arbitrary rules leads to the major problems of either being too general (e.g., false positives) or too specific (e.g., false negatives), leaving analysts frustrated and not effectively preventing intrusions and malicious activity. If the analysts want the false alarms to stop, they either need to make the rule more specific or deactivating it. Deactivating a rule is obviously not good and gives the attacker an advantage. Making the rule more specific is also not ideal, as it makes the rule fragile and more likely to be circumvented by trivial changes, also giving the attacker the advantage. Put simply, the properties of intelligence and adaptability are contained entirely within the human users.
Many attempts have been made over the years to design automated machine-learning algorithms to detect intrusions and attacks. Such systems fall into two general categories: misuse based and anomaly based. There is clearly no “silver bullet” algorithm. Rather, some attacks are easier to detect than others, some algorithms excel in some situations and fail in others. These algorithms do not find there way into production systems even though they possess utility because no algorithm solves the whole problem. Integration of each algorithm into a scalable and extensible platform that supports real-world production capabilities rather then academic proof-of-concept is an extremely non-trivial task.